@vtjnash, please confirm that this is the intended way to use this.

You seem to have closure captured interp instead of using the argument? The interp struct is commonly quite large, so that can increase memory usage quite a bit

The argument is the wrong interp

Yes, this way of defining a make_progress seems fine. There isn't really one right answer about how to code this, so base itself already uses probably 3 or 4 different patterns, depending on what kept the original code control flow seemed least distorted. I hadn't used the @isdefined trick, but it is essentially equivalent to the nextstate pattern I'd used for manual stackless state machine conversion

I don't think capturing the interp will give you the behavior you want. I think you might need to mutate sv instead of re-using sv with different interp

I'm not re-using sv with a different interp. sv here is an IRInterpretationState, which doesn't have an interp argument, so when the callback later gets scheduled, there's just some random interp in there.

sv claims to be a AbsIntState here? For IRInterpretationState currently it passes the interp here that was originally used to construct the IRInterpretationState, since everything is on the stack there and doesn't handle recursion

I think the behavior here is also probably fine, but that no other callback will be using the right interp, since none other are expecting the interp to be different from the one used to allocate the state object

and doesn't handle recursion

Doesn't handle recursion in Base ;).

Fair, I know almost nothing about this code, so I am reviewing without really knowing how this integrates. The current implementation in Base would potentially break here: https://github.com/JuliaLang/julia/blob/be401635fe02b28ce994e2e3cae0733d101f8927/base/compiler/ssair/irinterp.jl#L154
since it was not tracking if the return type changed to reschedule this instruction if it became part of cycle (I believe it should detect and @assert though if that attempts to happen)

Be careful not to closure capture any types, as your performance may suffer quite badly, but still just fast enough you won't notice (e.g. the sysimage could build still when I missed one of these cases, it just took several times longer)

So put it in a Ref{Any}?

Yeah, or a Core.Box equivalently. I'd found several places where we had a MethodMatch object that was needed anyways, so that also happened to work sometimes

The return type appears to be wrong here. The intended behavior appears to be returning result[] = primal_call[] in this case (

That's the function that calls this. Potentially it should be refactored to just do that, but I just wanted to only make the refactoring change.

The return type is required to be Future{CallMeta} though, or the caller's caller will be unhappy

The call-site there isn't updated yet. This function is called directly from DAECompiler and I adjusted the call-site there to work with Future{Union{Nothing, CallMeta}}

It is fine if it still branches on r!==nothing there, it will just be dead code now, as it appears you you must handle that case here now, instead of being able to handle it there